Web Survey Bibliography
Relevance & Research Question: Response latency measurement and eye tracking are two computer-assisted pretesting methods that may be particularly useful for evaluating Web questionnaires. In contrast to other techniques (e.g., expert reviews, qualitative interviews), both methods produce nonreactive and objective measures of behavior that are neither affected by the researcher (and the ways in which she tests the questions) nor by the research context. While previous studies have shown that longer response latencies and fixation times are indicative of problematic questions (Lenzner et al., 2010, 2011), little is known about the utility of the two methods (or measures) in the practical pretesting context (e.g., in testing draft questions). This study examines whether response latencies and fixation times are discriminative features to distinguish flawed from improved survey questions.
Methods & Data: In a laboratory experiment, respondents’ eye movements and response latencies were recorded while they were answering two versions of a Web questionnaire. One group (n=22) received a questionnaire including poorly worded questions and the other group (n=22) received the same questionnaire with improved question wordings. Given that response latencies and fixation times are highly individual, we computed the baseline fixation rate (eye tracking) and baseline reading rate (response latency) for every respondent from seven additional questions asked in the same Web survey. In the analyses, whenever the response or fixation times for a question exceeded respondents’ baseline by more than 15%, the question was deemed problematic. (The analyses were repeated with 10%, 20%, and 25% thresholds, but all conclusions remained unchanged).
Results: Fixation rate (eye tracking) was consistently more accurate than reading rate (response latency) in classifying the questions as flawed or improved. The overall accuracy of the fixation rate ranged from 60% to 85%, the accuracy of the reading rate from 43% to 70%. Also, the eye tracking measure resulted in considerably fewer misses (failures to detect problems) and fewer false alarms.
Added Value: This study suggests that fixation times and response latencies are potentially useful methods for pretesting (draft) Web questionnaires, albeit the level of accuracy with which they identify problematic questions is not yet satisfactory.
GOR Homepage (abstract) / (full text)
Web survey bibliography (4086)
- Is Pretesting Established Among Online Survey Tool Users?; 2012
- An Evaluation of Two Non-Reactive Web Questionnaire Pretesting Methods; 2012; Lenzner, T.
- Recommendations for implementing online surveys and simple experiments in social and behavioural research...; 2012; Hewson, C. M.
- High potential for mobile Web surveys: Findings from a survey representative for German Internet users...; 2012; Funke, F., Wachenfeld, A.
- A taxonomy of paradata for web surveys and computer assisted self interviewing (Casi); 2012; Callegaro, M.
- Can Social Media Research replace traditional research methods?; 2012; Faber, T., Einhorn, M., Hofmann, O., Loeffler, M.
- Bad Boy Matrix Question – Whatcha gonna do when they come for you?; 2012; Tress, F.
- Matrix vs. Single Question Formats in Web Surveys: Results from a large scale experiment; 2012; Klausch, L. T., de Leeuw, E. D., Hox, J., de Jongh, A., Roberts , A.
- Effects of Static versus Dynamic Formatting Instructions for Open-Ended Numerical Questions in Web Surveys...; 2012; Kunz, T., Fuchs, M.
- FamilyVote – Conducting online surveys with children and families; 2012; Geissler, H., Peeters, H.
- The influence of social desirability on data quality in face-to-face and web surveys; 2012; Keusch, F.
- Reducing the Threat of Sensitive Questions in Online Surveys; 2012; Couper, M. P.
- Using the Internet to Administer More Realistic Vignette Experiments; 2012; Caro, F. G., Ho, T., McFadden, D., Gottlieb, A. S., Yee, C., Chan, T.,
- Succinct Survey Measures of Web-Use Skills; 2012; Hargittai, E., Hsieh, Y. P.
- Data Use: Trying to bring the numbers alive ; 2012; Rung, J., Woll, B., Edstrom, C.
- The impact of two-stage highly interesting questions on completion rates and data quality in online...; 2012; Hansen, J. M., Smith, S. M.
- User models as revealed in web-based research services; 2012; Bodoff, D., Raban, D.
- When to Use What Research Design ; 2012; Vogt, W. P., Gardner, D. C., Haeffele, L. M.
- Assessing the Quality of Survey Data ; 2012; Blasius, J.
- Collecting, Managing, and Assessing Data Using Sample Surveys; 2012; Stopher, P.
- Designing and Doing Survey Research; 2012; Andres, L.
- The Impact of Visual Design in Survey Cover Letters on Response and Web Take-Up Rates; 2012; Mockovak, W.
- Validation of an online questionnaire measure of the relative reinforcing value of food; 2012; Reslan, S., Saules, K. K., Greenwald, M. K.
- Monitoring Validated Quality of Life Outcomes After Prostatectomy: Initial Description of Novel Online...; 2012; Sebrow, D., Lavery, H. J., Brajtbord, J. S., Hobbs, A., W., Levinson, A. W., Samadi, D. B.
- Who should supervise students during self-report interviews? A controlled experiment on response behavior...; 2012; Walser, S., Killias, M.
- Web-Based Recruiting for Health Research Using a Social Networking Site: An Exploratory Study; 2012; Fenner, Y., Garland, P., Moore, E. E., Jayasinghe, Y., Fletcher, A., Tabrizi, S. N., Gunasekaran, B.,...
- Exploring Animated Faces Scales in Web Surveys: Drawbacks and Prospects; 2012; Emde, M., Fuchs, M.
- Smart Surveys for Smart Phones: Exploring Various Approaches for Conducting Online Mobile Surveys via...; 2012; Buskirk, T. D., Andrus, C.
- Use of incentives and web-based administration for surveying student alcohol and substance use in an...; 2012; Onoye, J. M., Goebert, D., Nishimura, S.
- Reminders in Web-Based Data Collection: Increasing Response at the Price of Retention?; 2012; Goeritz, A., Crutzen, R.
- Does survey format influence self-disclosure on sensitive question items?; 2012; Kays, K., Gathercoal, K., Buhrow, W.
- Why semantic differentials in Web-based research should be made from visual analogue scales and not...; 2012; Funke, F., Reips, U.-D.
- Using Webinar Polls to Collect Online Survey Data: The Case of a Behavioral Finance Problem; 2012; Sahu, C.
- Non-Response Bias with a Web-Based Survey of College Students: Differences from a Classroom Survey About...; 2012; Wells, W., Cavanaugh, M. R., Nobles, M. R., Bouffard, Je. A.
- The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community...; 2012; Gajic, A., Cameron, D., Hurley, J.
- Multiple Sources of Nonobservation Error in Telephone Surveys: Coverage and Nonresponse; 2011; Peytchev, A.; Carley-Baxter, L. R.; Black, M. C.
- Online Questionnaires for Outbreak Investigations; 2011; Parry, A. E.; Johnson, D. R.; Byron-Gray, K.; Raupach, J. C. A.; McPherson, M.
- Inventory of published research: Response burden measurement and reduction in official business statistics...; 2011; Giesen, D. & Snijkers, G. (Eds.), Bavdaz, M., Bergstrom, Y., Gravem, D. F., Haraldsen, G., Hedlin, D...
- Effects of speeding on satisficing in Mixed-Mode Surveys; 2011; Bathelt, S., Bauknecht, J.
- Using Research-Based Practices to Increase Response Rates of Web-Based Surveys; 2011; Perkins, R. A.
- Using break-offs in web interviews for predicting web response in mixed mode surveys; 2011; Beukenhorst, D.
- Web panels in Slovenia; 2011; Lenar, J.
- Traditional and non-traditional treatments for autism spectrum disorder with seizures: an on-line survey...; 2011; Frye, R. E., Sreenivasula, S., Adams, J. B.
- Understanding the new digital divide—A typology of Internet users in Europe; 2011; Brandtzæg, P.B.; Heim, J.; Karahasanoviæ, A.
- Patients’ attitudes toward side effects of antidepressants: an Internet survey; 2011; Kikuchi, T., Uchida, H., Suzuki, T., Watanabe, K., Kashima, H.
- Web-based or paper-based surveys: a quandary?; 2011; Bennett, L., Sid Nair, C.
- Refining the Total Survey Error Perspective; 2011; Smith, T. W.
- ELIPSS: Étude Longitudinale par Internet Pour les Sciences Sociales; 2011; Legleye, S., Lesnard, L.
- Less questions, more data: Revitalizing the european currency in single source affluent audience measurement...; 2011; Hartman, H.
- Linking website exposure data to survey data: A single-source solution; 2011; Krahn, J., Landi, J., Melton, E.